KOBE-TH-11-08 



B° — B° Mixing in Gauge-Higgs Unification 



Yuki Adachi, Nobuaki Kurahashi*, Nobuhito Mam** and Kazuya Tanabe* 

Department of Sciences, Matsue College of Technology, Matsue 690-8518, Japan. 
* Department of Physics, Kobe University, Kobe 657-8501, Japan. 
** Department of Physics, and Research and Education Center for Natural Sciences, 
Keio University, Yokohama 223-8521, Japan. 



Abstract 

We discuss flavor mixing and resulting Flavor Changing Neutral Current (FCNC) in 
a five dimensional SU(3) co \ or <8> SU(3) <S> U'(l) gauge-Higgs unification. Flavor mixing is 
realized by the fact that the bulk and brane localized mass terms are not diagonalized 
simultaneously. As the concrete FCNC processes, we calculate the rate of B® - mixing 
and B° s -B° s mixing due to the exchange of non-zero Kaluza-Klein gluons at the tree level. 
We obtain a lower bound on the compactification scale of order O(TeV) by comparing 
our prediction on the mass difference of neutral B meson with the recent experimental 
data. 



1 Introduction 



In spite of the great success of the Standard Model (SM), the origin of electroweak gauge 
symmetry breaking is still unknown in particle physics. Though in the SM, Higgs boson 
is assumed to play a role for the symmetry breaking, it seems to have various theoretical 
problems such as the hierarchy problem and the presence of many theoretically unpre- 
dicted arbitrary coupling constants in its interactions. 

Gauge-Higgs unification (GHU) 1| is one of the fascinating scenarios beyond the SM. 
It provides a possible solution to the hierarchy problem without supersymmetry. In this 
scenario, Higgs boson in the SM is identified with the extra spatial components of the 
higher dimensional gauge fields. A remarkable fact is that the quantum correction to 
Higgs mass is UV-finite and calculable due to the higher dimensional gauge symmetry 
regardless of the non-renormalizability of the theory. This has opened up a new avenue 
to solve the hierarchy problem p] . The finiteness of the Higgs mass has been studied and 
verified in various models and types of compactification at one-loop leveQ |3| and even at 
the two loop level [5]. The fact that the Higgs boson is a part of gauge fields implies that 
Higgs interactions are restricted by gauge principle and may provide a possibility to solve 
the arbitrariness problem of Higgs interactions as well. 

From such point of view, it seems that the following issues are particularly important 
for the GHU to be phenomenologically viable. The first one is whether there are any 
characteristic prediction on the observables subject to precision tests. The second one is 
how CP violation is achieved since the Higgs interactions are given by gauge interactions 
with real couplings. The last one is how flavor mixing is generated since Yukawa coupling 
in GHU is given by gauge interactions which are universal for all flavors. 

As for the first issue, it will be desirable to find finite (UV-insensitive) and calculable 
observables, in spite of the fact that the theory is non-renormalizable and observables are 
very UV-sensitive in general. Works on the oblique electroweak parameters and fermion 
anomalous magnetic moment from such a viewpoint have been already done in the litera- 
ture [6-8 1 . The second issue has been addressed in our previous papers (9j[l0), where CP 
violation is claimed to be achieved spontaneously either by the VEV of the Higgs field or 
by the complex structure of the compactified extra space. 

In this paper, we focus on the remaining issue concerning the flavor physics in the 
GHU scenario. It is highly non-trivial problem to explain the variety of fermion masses 
and flavor mixings in this scenario, since the gauge interactions should be universal for all 
matter fields, while the flavor symmetry has to be broken eventually in order to distinguish 
each flavor and to realize their mixings. In our previous papers [11,12 , we addressed this 



issue and have clarified the mechanism to generate the flavor mixings by the interplay 
between bulk masses and the brane localized masses. 



1 For the case of gravity-gauge-Higgs unification, see [Ij. 
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Important point is that such introduced two types of mass terms generically may be 
flavor non-diagonal without contradicting with gauge invariance, which leads to the flavor 



mixing in the up- and down-types of Yukawa couplings 13 . We may start with the base 
where the bulk mass terms are diagonalized, since the bulk mass terms are written in the 
form of hermitian matrix, which may be diagonalized by suitable unitary transformations, 



keeping the kinetic and gauge interaction terms of fermions invariant 11,12 . Even in 
this base, however, the brane-localized mass terms still have off-diagonal elements in the 
flavor base in general. Namely, the fact that two types of fermion mass terms cannot be 
diagonalized simultaneously leads to physical flavor mixing. 

Once the flavor mixings are realized, it will be important to discuss flavor changing 
neutral current (FCNC) processes, which have been playing a crucial role for checking the 
viability of various new physics models, as is seen in the case of SUSY model. This issue 



was first discussed in 14 in the context of extra dimensions. Since our model reduces 
to the SM at low energies, there is no FCNC processes at the tree level with respect to 
the zero mode fields. However, it turns out that the exchange of non-zero Kaluza-Klein 
(KK) modes of gauge bosons causes FCNC at the tree level, though the rates of FCNC 
are suppressed by the inverse powers of the compactification scale ("decoupling") [IT 12 



The reason is the following. The gauge couplings of non-zero KK modes of gauge boson, 
whose mode functions are y-dependent, to zero mode fermions are no longer universal 
since the overlap integral of mode function of fermion and KK gauge boson depends on 
the bulk mass M different from flavor by flavor in general. 

In the previous papers, as typical processes of FCNC, we have calculated the K°-K° 
mixing and the D° - D° mixing amplitude at the tree level via non-zero KK gluon exchange 
and obtained the lower bounds for the compactification scale as the predictions of our 



model 11 12 . Interestingly, the obtained lower bounds of O(10) TeV were much milder 
than what we naively expect assuming that the amplitude is simply suppressed by the 
inverse powers of the compactification scale, say O(10 3 ) TeV. We pointed out the presence 
of suppression mechanism of the FCNC processes, which is operative for light fermions in 
the GHU model. In the analysis, we focused on the simplified two generation scheme in 
order to estimate the mass difference and the lower bound on the compactification scale. 

On the other hand, these suppression mechanism in the third generation containing 
top and bottom quarks does not work so strongly by the absence of bulk masses as we will 
discuss in the main text. Then it is expected that the dangerous large FCNC containing 
the third generation such as B° — B° mixing arises and more stringent constraints will 
be obtained. Thus it would be more desirable to discuss the FCNC process in the three 
generation scheme. 

In this paper, we discuss flavor mixings in the three generation model and especially 
consider the typical FCNC processes, i.e. B® — B% mixing and B Q S -B Q S mixing, which is 
caused by the mixing between down and bottom quarks or strange and bottom quarks. 
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We will calculate the dominant contribution to the B®-B® mixing and the B° s -B° s 
mixing at the tree level by the non-zero KK gluon exchange. The rate of the FCNC 
processes is suppressed by the small mixings between the third generation and lighter 
generations. Comparing the prediction of our model with the data, the lower bound on 
the compactification scale is obtained. 

This paper is organized as follows. After introducing our model in the next section, we 
summarize in section 3 how the flavor mixing is realized in the context of the gauge-Higgs 



unification, which was clarified and described in detail in our previous paper 11,12]. In 
section 4, as an application of the flavor mixing discussed in section 3, we calculate the 
mass difference of neutral 5-mesons caused by the B^ - B^ mixing and the B® - B® mixing 
via non-zero KK gluon exchange at the tree level. We also obtain the lower bound for 
the compactification scale by comparing the obtained result with the experimental data. 
Our conclusion is given in section 5. 

2 The Model 

The model we consider in this paper is a five dimensional (5D) SU (3) co i or <8> SU (3) © U'(l) 
GHU model compactified on an orbifold S jZ-i with a radius R of S . The three generation 
model is basically obtained by extending our previous model, but top quark mass cannot 
be incorporating as it stands. It is known that the fermion masses have an upper bound 
in GHU, 

m q < y/nMyy (m w : PU-boson mass j (2.1) 



where n is the number of indices of the representation the fermion belongs to [16]. Up- 
type quarks in our model belong to the totally symmetric tensor representation of SU(3), 
i.e. n = 2, in our two generation model. Thus, we should modify our model to obtain 
the correct top mass m t ~ 2Mw- Obviously, the simplest choice would be a 4-rank 
tensor representation. The representation of rank 4 of SU(3) are known to be 15, 24 



and 27 17 . We modify our model by using the smallest representation 15. Although it 
is still remaining a small gap between top and twice of PU-boson masses, it is attributed 
to the quantum correction of top Yukawa coupling. Focusing on the quark sector, we 
introduce three generations of bulk fermion in the 3, two generations of them in the 6 
and one generation of bulk fermion in the 15 dimensional representations of SU(3) gauge 
group, 

V>*(3) = Ql © cT (i= 1,2,3), (2.2a) 

V' < (6) = E 1 6 ©Q* 6 e« < (i = 1,2), (2.2b) 

V>(T5) = 0©Affi£ 15 ©Q 15 ffit (2.2c) 
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where all of the fermions are decomposed into those in the representations of SU (2) sub- 
group of SU (3) gauge group. An extra U'(l) is required for ij) (15) to fix the hypercharges. 
These sets of fermions contain ordinary quarks of the SM in the zero mode sector, i.e. Q\ 
and Qq (i = 1,2) corresponding to the first two generation quark doublets, Q 1 ^ 3 and Q15 
corresponding to the third generation quark doublet, and d l (i = 1,2,3), u l (i = 1,2), 
t corresponding to three generation down-type quark singlets, the first two generation 
up-type quark singlets, top quark singlet, respectively. ip l (6) have SU(2) triplet exotic 
states Eg and ip (15) also does SU{2) quintet, quartet, triplet exotic states 6, A and £15. 
The bulk Lagrangian is given by 

C = -\tt(F mn F mn ) - \B MN B MN - \ft(G MN G MN ) 
+ #(3) {ip 3 - M*e(j/)}^(3) + ft= 3 (3)ip 3 ^ i=3 (3) 
+ #(6) {ip 6 - M*e(</)}^(6) +^(15)i^(15) (2.3) 

where the gauge kinetic terms for SU(3), U'(\), SU(3) co \ OT and the covariant derivatives 
are 

Fmn = d M A N - d N A M - ig [A M , A N ] , (2.4a) 

Bmn = d M B N — d N B M , (2.4b) 

Gmn = 9mGn — 9nGm — ig s [Gm, G^~\ , (2.4c) 

p = r M (d M - tgA M - ig s G M ) , (2.4d) 

p' = r M {d M - igA M - ig' Bm - ig s G M ) ■ (2.4e) 

The gauge fields Am and Gm are written in a matrix form, e.g. Am = A a M ^- in terms 
of Gell-Mann matrices A a . It should be understood that Am in the covariant derivative 
Dm = dM — igAM — ig s GM ac ts properly depending on the representations of the fermions. 
M, N = 0, 1, 2, 3, 5 denotes indices of the bulk space-time. The five dimensional gamma 
matrices are given by F M = (7 M ,27 5 ) (// = 0, 1,2,3). g, g' and g s are 5D gauge coupling 
constants of SU(3), U'(l) and 577(3) co i or , respectively. M l (i = 1,2) are generation 
dependent bulk mass parameters of the first two generations of fermion accompanied by 
the sign function e(y). For the third generation, the bulk mass parameter should be taken 
to be zero to reproduce top quark mass. 

The periodic boundary condition is imposed along S 1 and Z 2 parity assignments are 



taken for gauge fields as 

A, (-y) = PA,(y)p-\ A y (-y) = -PA y {y)P~\ (2.5a) 
G,(-y) = GM, G y (-y) = -G y (y), (2.5b) 

B, {-y) = B„{y), B y (-y) = -B y (y) (2.5c) 



where the orbifolding matrix is defined as P = diag(— , — , +) and operated in the same 
way at the fixed points y — 0, irR. We can see that the gauge symmetry SU (3) is explicitly 
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broken to 577(2) x U(l) by the boundary conditions. The gauge fields with Z2 odd parity 
and even parity are expanded by use of mode functions, 

S n (y) = \ sin ^-y , CJy) = -== ■. cos ^-y, (2.6) 

respectively. 

The Z 2 parities of fermions are assigned for each component of the representations as 
follows: 

**(3) = {Qid+, +) + QU-, -) } © {4(- -) + 4(+, +) } (2 = 1,2,3), 
**(6) = {s i 6i (-, -) + su(+, +)} © {qi l (+, +) + qu(-, -)} 

©K(- -) + <(+,+)} (2 = 1,2) , 

= {e L (- -) + e fl (+, +)} © { a l (+, +) + a r (-, -)} 

© {£l5L(- ") + Si5fl(+, +)} © {Ql5L( + , +) + Ql5/?(-, -)} 
©{*£(--)+*«(+,+)}■ 

Thus a chiral theory is realized in the zero mode sector by Z2 orbifolding. 

The fermions are also expanded by an orthonormal set of mode functions. Here we 
will focus on the zero-mode sector necessary for the argument of flavor mixing. The zero 
mode sector of each component of and ^(15) are written in the following 

way. 

Ql = QizfM > dl = d Rfk(y) (^ = 1,2,3), (2.7a) 

= n R fk(y) , Ql = QkfM , « 4 = 4/t(y) (2 = 1,2) (2.7b) 

e = e R f R (y) , A = A L f L (y) , S 15 = E 1SR f R (y) , Q 15 = Qi 5L f L (y) , t = t R f R (y). 

(2.7c) 

The mode function for the zero mode of each chirality is given in pi: 



We notice that there are two left-handed quark doublets and Qql(Qi5l) per gener- 
ation in the zero mode sector, which are massless before electro-weak symmetry breaking. 
In the one generation case, for instance, one of two independent linear combinations of 
these doublets should correspond to the quark doublet in the SM, but the other one 
should be regarded as an exotic state. Moreover, having an exotic fermion £ 6 #, Eisij, A^ 
and Q R , we therefore introduce brane localized four dimensional Weyl spinors to form 
SU(2) x £7(1) invariant brane localized Dirac mass terms in order to remove these exotic 



massless fermions from the low-energy effective theory 13, 19 



£bLM — £rLM + ^BLM + ^BLM + -^BLM + ^BLM (2-9) 
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where for the first two generations, 



4Lm = / <kj V2^Rmll M 8{y - ttRM r (x, y)^ L {x) + (h.c.) 

— kR 



and for the third generation, 



"^BLM 



"^BLM 



ttR 

dyV2icRm^ M 5(y - irR)E 15R (x, y)E 15L (x) + (h.c.) 

-ttR 
ttR 

BLM* 5 \V) A L(x,y) A r(x) + (h.C.) , 



c 



m 

-ttR 
tvR 

l!lm / rf 2/v / 27rRmf LM 5(?/ - irR)Q R (x,y)Q L (x) + (h.c 



G 



and for three generations i = 1, 2, 3 



rQ 

"^BLM 



ttR 



where 



dy V2^RS{y)Q R {x)\r] ij Qi L {x,y) + \ i3 Q J L (x,y) \ + (h.c.) 

nR ( J 



Q L {x,y) = Q\ L (x,y) Qj L (x,y) Qi 5L (x,y) 



(2.10a) 



(2.10b) 
(2.10c) 
(2.10d) 



(2.10e) 



(2.11) 



Q R , S 615£ , A R and 0^ are the brane localized Weyl fermions of doublet, triplet, quartet 
and quintet of SU(2) respectively. The 3x3 matrices 77^, Ay and wiblms are mass 
parameters. These brane localized mass terms are introduced at opposite fixed points 
such that Q R , A R (E 6jl5 £, 0^) couples to Q3,6,i5L ; Ar (S 6,15/?, ©i?) localized on the brane 
at y — (y — ttR). Let us note that the matrices r]ij, Xij can be non-diagonal, which are 



the source of the flavor mixing 11 13 



3 Flavor mixing 



In the previous section we worked in the base where fermion bulk mass terms are written in 
a diagonal matrix in the generation space. Then Yukawa couplings as the gauge interaction 
of A y is completely diagonalized in the generation space. Thus flavor mixing does not 
happen in the bulk and the brane localized mass terms for the doublets Qsl and Qql (Qi5l) 
is expected to lead to the flavor mixing. We now discuss how the flavor mixing is realized 
in this model. 

First, we identify the SM quark doublet by diagonalizing the relevant brane localized 
mass term, 



ttR 



dyV2^R5(y)Q R (x)[ rj X } 



nR 



Q3L(x,y) 



DV2^RQ R {x)[r)f L (0) A/ L (0)] 



x) [ W-diag O3X3 ] 



Q3L(x) 

Ql{x) 

Qhl(x) 
Qsml(x) 

(3.1 
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where 



u 2 u 4 



Qhl(x) 
Qsml(x) 



U Q [vh(0) A/ x (0)] 



Qzl{x) 
Ql{x) 

U x U 3 
U 2 U 4 



U Q Q R (x) = Q' R {x) 



[ m diag 3X 3 ] • 



(3.2a) 



(3.2b) 



In eq. ( |3.1 ), ^/l(0) is an abbreviation of a 3 x 3 matrix whose element is given by 
r)ijfl(0), for instance. U 3 , U 4 are 3x3 matrices satisfying the unitarity condition 



u\u 3 



u\u A 



L 3x3, 



(3.3) 



which indicates how the quark doublets of the SM are contained in each of Q 3 l{x) and 
Q6,i5i( x ) an d compose a 6 x 6 unitary matrix together with U\, U2, which diagonalizes 
the brane localized mass matrix. The eigenstate Qn becomes massive and decouples from 
the low energy processes, while Qsm remains massless at this stage and is identified with 
the SM quark doublet. After this identification of the SM doublet, Yukawa couplings 
are read off from the higher dimensional gauge interaction of A y , whose zero mode is the 
Higgs field H(x): 



9a 
' 2 



'SML 



+ h.c. (3.4) 



where 04 = J—=- and the matrix W indicates the factor \fn in (|2.1|): 

W = diag(V2, V2,2) 



(3.5) 



and I R ^ is an overlap integral of mode functions of fermions with matrix elements 
{ a b.l )a — °» J * 



13 RL 



MOO) 
RL 



ttR 

dyflfk 

nR 



7TRM i 



smh(irRM i 
1 



(i = l,2) 
(< = 3) 



(3.6) 



which behaves as 2nRM l e~ nRAP for tcRM 1 ^> 1, thus realizing the hierarchical small 
quark masses without fine tuning of M l . We thus know that the matrices of Yukawa 
coupling yY„ and ^Y d are given as 



94 



94 



RL u 4 , 



(3.7) 



2 2 

These matrices are diagonalized by bi-unitary transformations as in the SM and Cabibbo- 
Kobayashi-Maskawa (CKM) matrix is defined in a usual way [18] . 



Y d = diag(m d , • • • ) = Vj R Y d V dL 



Y u = diag(m u , 



Vl R WY u V uL 



v< 



CKM 



VLVuL 



(3.8) 
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A 



where all the quark masses are normalized by the VF-boson mass as mj 
remarkable point is that the Yukawa couplings ^Y u and ^-Y^ are related through the 



mj_ 
M w ' 



unitarity condition eq. (3.3), on the contrary those are completely independent in the SM. 



For an illustrative purpose to confirm the mechanism of flavor mixing, we will see how 
the realistic quark masses and mixing are reproduced. Here we leave aside CP violation 
since the issue discussed in this paper is independent of it and assume that U 3 , £/ 4 are 



real. Let us notice that 3x3 matrices can be parametrized because of (3.3) without 
loss of generality as 



U4 — R u 



a\ 
a 2 







«3 



U, = R n 



1 — a 


















(3.9) 



where R u and R<i are arbitrary 3x3 rotation matrices parametrized as 



Ru 



Rri 



1 


1 













cos 9' 2 sin 9' 2 
— sin8' cos #Q 



cos 6'o sin 9'n 







1 







-sin 9L cos 





cos 9 2 
—sin 9 2 





sin 9 2 

cos 9 2 



cos 6*3 


—sin 9s 



sin 9 3 

1 

cos 9? 



cos9[ -sm9[ 

sin0i cosfl; 

1 

cos 9i —sin 9i 

sin 9 1 cos 9 1 

1 



(3.10a) 



(3.10b) 



Actually the most general forms of U 3 and U4 have a common orthogonal matrix multiplied 
from the right, which can be however eliminated by suitable unitary transformations 
among the members of Qsml(x). 

Now physical observables rh u , m c , rh t , rha, rh s , rhb and the angles of the CKM matrix 
are expressed in terms of a^, ) and 6 rotation angles in R u and Rd- Note that 

our theory has 2 free parameters which cannot be determined by the observables since 9 
physical observables are written in terms of 11 parameters. 



As we have discussed in the previous paper 12 , if the large mixings between the 1-3 



and 2-3 generations are introduced then the top quark mass decreases from 160 GeV 
~ 2M\y- Thus, we expect that the mixing angles between the third generation and the 
first two generations are considered to be small to keep m t ~ 2M\y. Also the relation 
between the masses of top and bottom quark mf + (2m{,) 2 = (2Mw) 2 for the M3 = 
holds and we must choose 03 ~ 1. It implies that the rotation angles 9' 2 , 9' 3 , 9 2 , 9 3 and 
parameter ^/l — a| should be small, and also other 6 parameters a\, a 2 , &i, b 2 and 9[, 9i 



should take values close to those of 2 generation model 12 

Actually, for the case of R u = 1 3X 3 where the up type quark mixings vanish, this 
case gives almost the most stringent lower bound from K° - K° mixing for example, these 
parameters are numerically found as 



b\ w 4.355 x 10" 



sin 6*i 



-2.587 x 10 



-2 
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a\ « 0.9887 , h\ « 1.302 x 10" 4 , sin# 2 ~ 2.224 x 10~ 2 . (3.11) 
a 2 3 & 0.9966 sin fl 3 « 2.112 x 10~ 4 



Also, for the another case of Rd = I3X3 where 
parameters are numerically found as 

a 2 « 0.0650 6 2 « 3.973 x 10" 9 

a\ « 0.9931 , ^ w 2.235 x 10" 4 
« 0.9966 



the down type quark mixings vanish, these 
sinfli « 0.6704 

sin #2 « -3.936 x 10~ 2 . (3.12) 
sin# 3 « 1.773 x 10~ 2 



These two results show that the mixing angles 9%, #3, 9' 2 , 9' 3 <C 1 and it is completely 
consistent with the above argument. 



4 B° — B° mixing 



In this section, we apply the results of the previous section to representative FCNC 
processes, B®-B® mixing and B®-B® mixing responsible for the mass difference of two 
neutral B mesons J3 

We focus on the FCNC processes of zero-mode down-type quarks due to gauge boson 
exchange at the tree level. First let us consider the processes with the exchange of zero 
mode gauge bosons. If such type of diagrams exist with a sizable magnitude, it will easily 
spoil the viability of the model. 

Concerning the Z-boson exchange, it is in principle possible to occur the tree-level 
FCNC. Since the mode function of the zero-mode gauge boson is y- independent, the 
overlap integral of mode functions is generation independent. Thus the gauge coupling of 
zero mode gauge boson depends on only the relevant quantum numbers such as the third 
component of weak isospin 1%. Therefore the condition proposed by Glashow- Weinberg 



15 to guarantee natural flavor conservation for the theories of 4D space-time is relevant. 

Although there are right-handed down-type quarks belonging to different representa- 
tions in our model, for example, the SU(2) singlet d,R in if>(3) and one of components 
of the triplet in ^(6) or t/>(15), these are known to have the same quantum number 



^3 = 0, and thus the Glashow- Weinberg condition is satisfied in this sector 11 . However, 
the quintet Qr in ^(15) also contains the right-handed down-type quark, and this has 
the different quantum number I3 = 1 from that of d R belonging to 

What is worse, the quartet in ip(15) contains left-handed down-type quark with the 
different quantum number J 3 = | from that of d l L belonging to the doublet Ql in if) (3), 
t/>(6) or t/>(15) with the quantum number J 3 = —5. Thus, the condition of Glashow- 
Weinberg is not satisfied in the down-type quark sector and FCNC process due to the 



2 For the studies of B° - B° mixing in other new physics models, see for instance 21 
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exchange of the zero mode Z-boson arises at the tree levelj^] However, the quintet Or 
(quartet A^) is an exotic fermion and acquires large 577(2) invariant brane mass. Thus 
the mixing between d R (d l L ) and Qr (A L ) is inversely suppressed by the power of ttiblm 
and the FCNC vertex of Z-boson can be safely neglected. We may say that the condition 
of Glashow- Weinberg is satisfied in a good approximation in the processes via the zero 
mode gauge boson exchange. 

One may worry that Z' gauge boson exchange give rise to FCNC processes since an 
extra U(l) gauge symmetry is indispensable for getting a realistic Weinberg angle. Note 
that the extra £7(1) gauge symmetry is explicitly broken by an anomaly and the gauge 
boson of the extra U(l) gauge symmetry acquire a mass of the cutoff scale order. In 
our model, the cutoff scale is a 5D Planck scale which is larger than the intermediate 
scale 10 13 GeV. Therefore, the FCNC effects by Z' gauge boson exchange can be safely 
neglected comparing to the process by non-zero KK gluon exchanges considered later. 

Hence, the remaining possibility is the process via the exchange of non-zero KK gauge 
bosons. In this case, the mode functions of KK gauge bosons are y-dependent and their 
couplings to fermions are no longer universal because of non-degenerate bulk masses, even 
if the condition of Glashow- Weinberg is met. 

Therefore, such progresses lead to FCNC at the tree level. In our previous papers 



11 



12 , we have calculated K°-K° mixing and D°-D° mixing via the non-zero KK 
gluon exchange at the tree level and obtained a lower bound of the compactification scale 
as the prediction of our model. Along the same line of the argument as in our previous 
papers, we here study B^-B^ and B®-B® mixings in the down-type quark sector by 
the non-zero KK gluon exchange at the tree level as the dominant contribution to these 
FCNC processes. 

For such purpose, let us derive the four dimensional effective QCD interaction vertices 
for the zero modes of down-type quarks relevant for our calculation: 

+ |G^^AV4(-l) n {^(^/S°^3 + U\l$fu,)v dL }^ . (4.1) 

where Irr and i^^^ are overlap integrals relevant for gauge interaction, 

r«°»o> 1 r".f«^...".. 1 (2flM<) 2 (-l)»e MM '-l 



fm " = 7mJ d l {f>R) ms R y= 7m (2RM¥T^ e^-i ■ < 42 > 



Ti(OnO) _ T i(0n0) 
LL — 1 RR 



M l -> —M* 



r i(0n0) 
RR 

since the chirality exchange corresponds to the exchange of two fixed points. We can see 



from (4.1) that the FCNC appears in the couplings of non-zero KK gluons due to the 



3 The FCNC due to the exchanges of zero-mode photon and gluon trivially vanish because the fermions 
of our interest have the same electric charge and color. 
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d L b L d L b L d R b R 
* 3 * * 3 * * 3 * 

9 r< a(n) S n a(n) S n a{n) 



d R b L d L b R d R 

(i) LR type (ii) LL type (iii) RR type 

Figure 1: The diagrams of B®-B^ mixing via KK gluon exchange. Those of B®-B® mixing 
via KK gluon exchange are obtained by the replacements d s. 

fact that I RR °^ is not proportional to the unit matrix in the generation space, while the 
coupling of the zero mode gluon is flavor conserving, as we expected. 

The Feynman rules necessary for the calculation of B^-B® mixing can be read off 



from (4.1 ). 



Qd(n) 



g b R = 9 i{v^V dR ) 3 X^R, (4.4a) 



d R 

»• — 

Qa(ri) 



d L _J b L = %{-l) n {vL(ul&U a + Utl^U A )v dL }^L , 

(4.4b) 



A 4 ^ v XX ' 



rf 1 ( \ 

Snn'Sab k2 _ M2 { 't Hooft-Feynman gauge J . (4.4c) 



Those for B Q S -B Q S mixing are easily obtained by the replacements d -H- s and 31 -H- 32 in 
the matrix element of the vertices. The non-zero KK gluon exchange diagrams providing 
the dominant contribution to the process of B^-B® and B° s -B° s mixing are depicted in 
Fig.0 

By noting the fact k 2 <C (-|) 2 for n^O being the mass of ra-th KK gluon and being 
internal momentum, the contributions from each type diagram of the B^-B® mixing 
in Fig.[TJ are written in the form of effective four- Fermi lagrangian obtained by use of 
Feynman rules listed above, 

di b L 
> * > 



n=l 
-l) n 



n 2 l "~ \ ~ / J 31 V """ / 31 

(4.5a) 



{vL(uli£?u a + utiSfu 4 )v dL } 3 lvJ R i^ v dR 
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d L 



00 2 1 

^ n=l c 



die 



^{^(c/t/srvs + uu^y^i , (4.5b) 



2 1 1 9 

^ n=l c 



(4.5c) 

Similarly, those for B° s -B° s mixing are obtained by the replacements d -H- s and 31 -H- 32 
in the matrix element of the vertices. The sum over the integer n is convergent and 



the coefficients of the effective lagrangian (4.5) for the B®-B® mixing and (4.5) after 



the replacements of d •<-)■ s and 31 -H- 32 for the B° s -B® mixing are suppressed by the 
compactification scale as 1/M C 2 = R 2 . 

Comparing the results with the experimental data, we can estimate a lower bound on 
the compactification scale. The most general effective Hamiltonian for AB = 2 processes 
due to some "new physics" at a high scale Anp 3> Mw can be written as follows; 



L NP \i=l i=l / 



^ e A /= 2 = ^- ( y m\ + y m i (4.6) 



where the 4- Fermi operators relevant for B\ - B\ mixing are given as, 

Qi = dt^bldlYbl , Ql = d a R b a L d R b{ , Qt = d a X4b a L , 

Qt = , Qi = d%b{d{b R (4.7a) 

and for B® - B® mixing, 

Qi = , Q s 2 = , Qs = s a R b{s R bl , 

Ql = s a R bls P L b R , Ql = s a X4b R . (4.7b) 

Indices a, (5 stand for the color degrees of freedom. The operators (Ji,2,3 are obtained 
from the Qi,2,3 by the chirality exchange L -H- R. Since the SM contribution is poorly 
known, we can get the constraint on the new physics directly from the experimental data 
assuming that there is no accidental cancellation between the contributions of the SM and 
new physics. If we assume one of these possible operators gives dominant contribution 
to the mixing, each coefficient is independently constrained as follows, with the same 
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constraints for z\ as those for z\ (i = 1, 2, 3) |20|; 



zf\ < 2.3 x 10 



-5 



A 



NP 



ITeV 



z$\ < 7.2 x 10" 



_7 / Anp 



ITeV 



4\ < 2.8 x 10 



-6 



A 



NP 



ITeV 



2 



< 2.1 x 10 



-7 



A 



NP 



ITeV 



zf| < 6.0 x 10 



-7 



A 



NP 



ITeV 



2:, 



< 1.1 x 10 



< 5.6 x 10" 



-3 



A 



NP 



ITeV 



A 



NP 



ITeV 



< 2.1 x 10 



-1 



A 



NP 



ITeV 



(4J 



< 1.6 x 10 



-5 



A 



NP 



ITeV 



< 4.5 x 10 



-5 



A 



NP 



ITeV 



where the new physics scale Anp is regarded as the compactification scale in our case. All 
we have to do is to represent (4.5) and its replacements d -H- s and 31 -H- 32 of (4.5) by 
use of (4.7) and to utilize these constraints (4.8). 

We can rewrite the each type effective lagrangian for B® — B% mixing (4.5) in terms of 
effective Hamiltonian by using the Fierz transformation and the completeness condition 
for Gell-Mann matrices; 



H 



AB=2 
eS.LL 



dnd 
1 



zfQ 



n 



AB=2 



zfQf 



AB=2 _ z tQt + z tQ 



H 



Rr 



where 



zd 



~3~ 

Srra;. 



00 

n=l 
00 1 



31 



(4.9) 

(4.10a) 
(4.10b) 



n=l 
00 



2. = — 87ra; 



( _1 ) n /T/t / , r rtr(*»°) 



n=l 



31 



T/t r(0"0) T/ 



(4.10c) 



-7TQ!. 



**£ ^{^(^^3 + c/i/r , ^)^} 3i (vj R /r > K,«) 3i . (4.iod> 

n=l 



The four-dimensional a s is defined by 



a. 



1 9t 



(4.11) 

47T 277^477 V ' 

The constant a s should be estimated at the scale //& = mj = 4.6 GeV where the AB = 2 
processes are actually measured 20 . So we have to take into account the renormalization 
group effect from the weak scale down to ^ 



23 Mz 

a~ l (m b ) = a~ l (M z ) - — In 

6rr nib 
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a s (m b ) « 0.207 (4.12) 



where a s (M z ) « 0.1184 has been put |22]. 

Similarly, the each type effective Hamiltonian for B° s -B° s mixing are respectively 



rewritten by replacements d -H- s and 31 <-> 32 of (4.9) and (4.10). 



Combining these results, we obtain the lower bounds for the compactification scale 



from the constraint (4.8). First let us assume that only one of the three types of diagrams 
(LL, RR, LR) gives dominant contribution to the mixing. Then we get lower bound on 
the compactification scale by use of the upper bound on the relevant coefficients zf, z\ 



and z\ given in (4.8) in the unit of TeV: 



LL : 



RR : 



LR 



_ > 
R ~ 

i> 
R ~ 

I> 
R ~ 





zf 




2.3 x 10- 5 




zd 
z l 




2.3 x 10- 5 




~d 

z A 





2.1 x 10" 7 



[TeV] 



LL : 



RR : 



LR 



_ > 
R ~ 

I> 
R ~ 

I> 
R ~ 



/ 






1.1 X lO- 3 


/ 






1.1 X 10- 3 


/ 







1.6 x 10" 5 



[TeV] 
[TeV] 
[TeV] 



(4.13) 



Let us note that LR type diagrams yield both of Q\ and Ql operators as is seen in (4.9). 
We can however safely ignore the contribution of Ql to the mixing, because the coefficients 



of the operator (4.10d) are smaller than that of Ql and also because the magnitude of the 



hadronic matrix element of Q\ is known to be greater than that of Q|, as the constraint 



for z\ is more severe that that for Q\ in (4.8). This is why we used the constraint for z\ 
alone to get the lower bound for the case of LR type diagrams. 

Since there is no bulk mass of third generation in this model, the "GIM-like" sup- 
pression mechanism from the large bulk masses which acts much more severe on the 
contribution of the LR type diagram [12] does not occur. Thus the contribution of the 
LR-type diagram is not expected to be smaller than those of the LL and RR diagram 
in general. Actually, for the case of R u = 1^x3 m (3.9), which gives almost the most 
stringent lower bound from K°-K° mixing, the LR type contribution is dominant for 
B® - B° s mixing while the LL type contribution is dominant for B d - B^ mixing; 



R~ x > 1.71 [TeV] for B° d -B° d mixing 
iT 1 > 2.54 [TeV] for B° s - B° s mixing 



(4.14a) 
(4.14b) 



In the second case Rd = ls X 3, the contributions from the LR and RR type diagram 
vanish. This is because the down-type Yukawa coupling becomes diagonal: VdL = VdR = 
lsx3i namely the mixings in the down quark sector disappear. Note, however, that the 
lower bound obtained from the LL type contribution, which does not vanish even though 
VdL = 13x3- Actually, we obtain the lower bound on i? -1 for Rd = I3X3; 



Rr 1 > 0.92 [TeV] for B° d -B° d mixing 



(4.15a) 
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R~ x > 1.79 [TeV] for B° s -B° s mixing . (4.15b) 

This is because V u l relevant for up-type quark mixing also contributes to the left-handed 
FCNC current. Namely, because of the mixing between Q 3 l and Qql (Qi5l), U± also 



contributes to the FCNC vertex (4.4b) Thus even in the case of VdL = 13x3 we get a 
meaningful lower bound on M c 

A comment is given. The obtained lower bounds are smaller than what we naively 
expect assuming that the tree level diagram relevant for the FCNC process is simply 
suppressed by 1/M C 2 (20) ; 

M c > C(l0 3 ) [TeV] for B° d - B° d mixing , (4.16a) 
M c > C(l0 2 ) [TeV] for B° s - B° s mixing . (4.16b) 

which is much more stringent than the lower bound we obtained, in spite of the absence of 
the suppression by the large bulk masses. The obtained lower bounds also are milder than 
those from K° — K° and D° — D° mixings. This apparent discrepancy may be attributed 
to the very small mixing between the third generation and the first two generations. 

5 Summary 

In this paper, we have discussed the flavor mixing and the resulting FCNC processes in the 
framework of five dimensional SU(3) co i or ®SU(3)®U'(l) gauge-Higgs unification scenario. 
As the concrete FCNC processes, we have calculated the contributions to B d — B® and 
B® — B° s mixings by the non-zero KK gluon exchange at the tree level in the light of the 
recent progress in the measurements of B° — B° mixing. For the processes with respect to 
the third generation, the "GIM-like" suppression mechanism, which is operative for the 
light first two generation quarks, does not work since their bulk mass has to be vanished 
to realize top quark mass. Therefore, we can anticipate large FCNC effects to arise and 
we are likely to obtain strong constraints for B-physics. The prediction of our model is 
that the lower bounds of compactification scale have been found to be of order O (TeV) 
which is milder than those obtained from our study of K° — K° and D° — D° mixings 
in our previous paper 11 12 and from a naive expectation (~ 1000 TeV) where the 
dimension six operator is simply suppressed by 1/M C 2 in spite of the absence of the GIM- 
like suppression by the large bulk masses. This is because the smallness of the mixings 
between 1-3 and 2-3 generations, i.e. 92,03,O' 2 ,9' 3 1. In our model, they should be 
small to reproduce the realistic top quark mass ~ 2M\y, and then the induced AB = 2 
effective hamiltonian are strongly suppressed. Thus the lower bound of compactification 
scale becomes small. 
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